<!DOCTYPE html>
<html lang="en">
    <head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta name="format-detection" content="telephone=no" />

  <title>
    52|案例篇：服务吞吐量下降很厉害，怎么分析？ | 迪克猪的博客
  </title>

  
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
  <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
  <link rel="manifest" href="/manifest.json" />
  <meta name="theme-color" content="#ffffff" />

  
  <link
    rel="stylesheet"
    href="https://unpkg.com/modern-normalize@0.6.0/modern-normalize.css"
  />

  
  
  
  
  <link rel="stylesheet" href="https://zsy619.github.io/style.min.f7761d111b74dd5c07f0111decee92938c12abc42e0fd319e1a07483e248b54e.css" integrity="sha256-93YdERt03VwH8BEd7O6Sk4wSq8QuD9MZ4aB0g&#43;JItU4=" />

  
  
    
  
</head>

    <body>
        <header id="header">
  <div class="header_container">
    <h1 class="sitetitle">
      <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a>
    </h1>
    <nav class="navbar">
      <ul>
        <li><a href="https://zsy619.github.io">Home</a></li>
        
          <li>
            <a href="/post/">
              
              <span>LINUX性能优化</span>
            </a>
          </li>
        
          <li>
            <a href="/csapp/">
              
              <span>深入理解计算机系统_第三版</span>
            </a>
          </li>
        
          <li>
            <a href="/golang/">
              
              <span>golang</span>
            </a>
          </li>
        
          <li>
            <a href="/docker/">
              
              <span>容器</span>
            </a>
          </li>
        
          <li>
            <a href="/flutter/">
              
              <span>Flutter</span>
            </a>
          </li>
        
          <li>
            <a href="/know/">
              
              <span>知识点</span>
            </a>
          </li>
        
          <li>
            <a href="/categories/">
              
              <span>目录</span>
            </a>
          </li>
        
          <li>
            <a href="/about/">
              
              <span>关于</span>
            </a>
          </li>
        
        <li class="hide-sm"><a href="https://zsy619.github.io/index.xml" type="application/rss+xml">RSS</a></li>
      </ul>
    </nav>
  </div>
</header>

        
<section id="main">
  <article class="post content">
    <h2 class="title">52|案例篇：服务吞吐量下降很厉害，怎么分析？</h2>
    <div class="post_content">
      <ul>
<li>使用 perf 配合火焰图寻找热点函数，是一个比较通用的性能定位方法，在很多场景中都可以使用。</li>
<li>如果这仍满足不了你的要求，那么在新版的内核中，eBPF 和 BCC 是最灵活的动态追踪方法。</li>
<li>而在旧版本内核，特别是在 RHEL 系统中，由于 eBPF 支持受限，SystemTap 和 ftrace 往往是更好的选择。</li>
</ul>
<h3 id="连接数优化">连接数优化</h3>
<p>要查看 TCP 连接数的汇总情况，首选工具自然是 ss 命令。</p>
<h3 id="工作进程优化">工作进程优化</h3>
<p>执行下面的 docker 命令，查询 Nginx 容器日志就知道了：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ docker logs nginx --tail <span style="color:#ae81ff">3</span>
192.168.0.2 - - <span style="color:#f92672">[</span>15/Mar/2019:2243:27 +0000<span style="color:#f92672">]</span> <span style="color:#e6db74">&#34;GET / HTTP/1.1&#34;</span> <span style="color:#ae81ff">499</span> <span style="color:#ae81ff">0</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span>
192.168.0.2 - - <span style="color:#f92672">[</span>15/Mar/2019:22:43:27 +0000<span style="color:#f92672">]</span> <span style="color:#e6db74">&#34;GET / HTTP/1.1&#34;</span> <span style="color:#ae81ff">499</span> <span style="color:#ae81ff">0</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span>
192.168.0.2 - - <span style="color:#f92672">[</span>15/Mar/2019:22:43:27 +0000<span style="color:#f92672">]</span> <span style="color:#e6db74">&#34;GET / HTTP/1.1&#34;</span> <span style="color:#ae81ff">499</span> <span style="color:#ae81ff">0</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span> <span style="color:#e6db74">&#34;-&#34;</span>

</code></pre></div><h3 id="套接字优化">套接字优化</h3>
<p>然后回到终端一中，观察有没有发生套接字的丢包现象：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
<span style="color:#75715e"># 只关注套接字统计</span>
$ netstat -s | grep socket
    <span style="color:#ae81ff">73</span> resets received <span style="color:#66d9ef">for</span> embryonic SYN_RECV sockets
    <span style="color:#ae81ff">308582</span> TCP sockets finished time wait in fast timer
    <span style="color:#ae81ff">8</span> delayed acks further delayed because of locked socket
    <span style="color:#ae81ff">290566</span> times the listen queue of a socket overflowed
    <span style="color:#ae81ff">290566</span> SYNs to LISTEN sockets dropped

<span style="color:#75715e"># 稍等一会，再次运行</span>
$ netstat -s | grep socket
    <span style="color:#ae81ff">73</span> resets received <span style="color:#66d9ef">for</span> embryonic SYN_RECV sockets
    <span style="color:#ae81ff">314722</span> TCP sockets finished time wait in fast timer
    <span style="color:#ae81ff">8</span> delayed acks further delayed because of locked socket
    <span style="color:#ae81ff">344440</span> times the listen queue of a socket overflowed
    <span style="color:#ae81ff">344440</span> SYNs to LISTEN sockets dropped

</code></pre></div><p>可以执行下面的命令，查看套接字的队列大小：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ ss -ltnp
State     Recv-Q     Send-Q            Local Address:Port            Peer Address:Port
LISTEN    <span style="color:#ae81ff">10</span>         <span style="color:#ae81ff">10</span>                      0.0.0.0:80                   0.0.0.0:*         users:<span style="color:#f92672">((</span><span style="color:#e6db74">&#34;nginx&#34;</span>,pid<span style="color:#f92672">=</span>10491,fd<span style="color:#f92672">=</span>6<span style="color:#f92672">)</span>,<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;nginx&#34;</span>,pid<span style="color:#f92672">=</span>10490,fd<span style="color:#f92672">=</span>6<span style="color:#f92672">)</span>,<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;nginx&#34;</span>,pid<span style="color:#f92672">=</span>10487,fd<span style="color:#f92672">=</span>6<span style="color:#f92672">))</span>
LISTEN    <span style="color:#ae81ff">7</span>          <span style="color:#ae81ff">10</span>                            *:9000                       *:*         users:<span style="color:#f92672">((</span><span style="color:#e6db74">&#34;php-fpm&#34;</span>,pid<span style="color:#f92672">=</span>11084,fd<span style="color:#f92672">=</span>9<span style="color:#f92672">)</span>,...,<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;php-fpm&#34;</span>,pid<span style="color:#f92672">=</span>10529,fd<span style="color:#f92672">=</span>7<span style="color:#f92672">))</span>

</code></pre></div><p>执行下面的命令，分别查询 Nginx 和内核选项对监听队列长度的配置：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
<span style="color:#75715e"># 查询nginx监听队列长度配置</span>
$ docker exec nginx cat /etc/nginx/nginx.conf | grep backlog
        listen       <span style="color:#ae81ff">80</span>  backlog<span style="color:#f92672">=</span>10;

<span style="color:#75715e"># 查询php-fpm监听队列长度</span>
$ docker exec phpfpm cat /opt/bitnami/php/etc/php-fpm.d/www.conf | grep backlog
; Set listen<span style="color:#f92672">(</span>2<span style="color:#f92672">)</span> backlog.
;listen.backlog <span style="color:#f92672">=</span> <span style="color:#ae81ff">511</span>

<span style="color:#75715e"># somaxconn是系统级套接字监听队列上限</span>
$ sysctl net.core.somaxconn
net.core.somaxconn <span style="color:#f92672">=</span> <span style="color:#ae81ff">10</span>

</code></pre></div><h3 id="端口号优化">端口号优化</h3>
<p>执行下面的命令，就可以查询系统配置的临时端口号范围：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range<span style="color:#f92672">=</span><span style="color:#ae81ff">20000</span> <span style="color:#ae81ff">20050</span>

</code></pre></div><h3 id="火焰图">火焰图</h3>
<p>执行 perf 和 flamegraph 脚本，生成火焰图：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
<span style="color:#75715e"># 执行perf记录事件</span>
$ perf record -g

<span style="color:#75715e"># 切换到FlameGraph安装路径执行下面的命令生成火焰图</span>
$ perf script -i ~/perf.data | ./stackcollapse-perf.pl --all | ./flamegraph.pl &gt; nginx.svg

</code></pre></div><p><img src="../../images/20200714-1818-20.png" alt=""></p>
<h3 id="小结">小结</h3>
<p>实际上，分析性能瓶颈，最核心的也正是掌握运用这些原理。</p>
<ul>
<li>首先，利用各种性能工具，收集想要的性能指标，从而清楚系统和应用程序的运行状态；</li>
<li>其次，拿目前状态跟系统原理进行比较，不一致的地方，就是我们要重点分析的对象。</li>
</ul>
<p>从这个角度出发，再进一步借助 perf、火焰图、bcc 等动态追踪工具，找出热点函数，就可以定位瓶颈的来源，确定相应的优化方法。</p>

    </div>
    <div class="info post_meta">
      <time datetime=2020-07-14T18:12:19&#43;0800 class="date">Tuesday, July 14, 2020</time>
      
        <ul class="tags">
        
          <li> <a href="https://zsy619.github.io/tags/linux%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96">linux性能优化</a> </li>
        
          <li> <a href="https://zsy619.github.io/tags/linux">linux</a> </li>
        
        </ul>
      
      
    </div>
    <div class="clearfix"></div>
  </article>
  
    <div class="other_posts">
      
      <a href="https://zsy619.github.io/post/51%E6%A1%88%E4%BE%8B%E7%AF%87%E5%8A%A8%E6%80%81%E8%BF%BD%E8%B8%AA%E6%80%8E%E4%B9%88%E7%94%A8%E4%B8%8B/" class="prev">51|案例篇：动态追踪怎么用？（下）</a>
      
      
      <a href="https://zsy619.github.io/post/53%E5%A5%97%E8%B7%AF%E7%AF%87%E7%B3%BB%E7%BB%9F%E7%9B%91%E6%8E%A7%E7%9A%84%E7%BB%BC%E5%90%88%E6%80%9D%E8%B7%AF/" class="next">53|套路篇：系统监控的综合思路</a>
      
    </div>
    <aside id="comments">
</aside>

  
</section>

        <a id="back_to_top" title="Go To Top" href="#">
  <span>
    <svg viewBox="0 0 24 24">
      <path fill="none" d="M0 0h24v24H0z"></path>
      <path d="M12 2L4.5 20.29l.71.71L12 18l6.79 3 .71-.71z"></path>
    </svg>
  </span>
</a>

        <footer id="footer">
  <p>
    <span>&copy; 2021 <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a> </span>
    <span>Built with <a rel="nofollow" target="_blank" href="https://gohugo.io">Hugo</a></span>
    <span>Theme by <a rel="nofollow" target="_blank" href="https://github.com/wayjam/hugo-theme-mixedpaper">WayJam</a></span>
  </p>

  <script src="https://zsy619.github.io/js/main.min.8b182175f5874aeed0acc0979345c98d4bde22208ec4f36cc1d6e3102acb4b10.js" integrity="sha256-ixghdfWHSu7QrMCXk0XJjUveIiCOxPNswdbjECrLSxA=" crossorigin="anonymous" async></script>
</footer>

    </body>
</html>
